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Remarks 

Claims 1-5 are pending in the application. In response to Applicants 1 16 March 2005 
amendment and request for continued examination, the previous rejections of the claims over Rice 
(U.S. Patent No. 6,310,619) were withdrawn and new grounds for rejection have been asserted. 
Through the arguments below that traverse the new grounds for rejection, the patentability of the 
presently claimed invention is established. 

I. Objections to the Drawings 

The Office Action Summary (Form PTOL-326) indicates that the drawings filed on 28 
April 2001 have been objected to by the Examiner, but the Detailed Action does not describe the 
deficiencies found in the drawings. Applicants respectfully note that formal drawings were 
submitted in this case on 25 July 2001. 

II. Claim Rejections Based Upon Cited Reference 

(a) Claims 3-5 were rejected under 35 U.S.C. § 102(e) as anticipated by Ross et al. (U.S> 
Patent No. 6,608,628, hereinafter "Ross"), and claims 1-2 were rejected under 35 U.S.C. § 103(a) 
as obvious over Ross. 

(b) Applicants note, as a first matter, that Ross was issued on 19 August 2003, after the 28 
April 2001 filing date of the present application, and after the 28 April 2000 filing date of the 
provisional application (Ser. No. 60/200,407) to which the present application claims priority. 
Thus, Applicants reserve the right to explore the possibility of "swearing behind" Ross's earliest 
effective filing date of 06 November 1998 in an affidavit under 37 CFR §1.131. 

(c) The presently claimed invention relates to software development of general haptic virtual 
environments, which can include creation or importation of a three-dimensional graphical virtual 
environments. The presently claimed invention (hereinafter, the" G2H" system or invention) 
permits generation of a user selected size and shape of a three-dimensional model to be used as a 
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proxy in order to represent the haptic device in the computational virtual space. This is 
important since otherwise there is no way of seeing what is being touched in the physical virtual 
device workspace by the haptic device. This is a crucial mapping of the correspondence needed 
between the physical virtual device workspace in which the haptic device is physically moved 
and the computational virtual space containing the virtual scene with virtual objects. From 
software development point of view this means that the graphical representation of the cursor 
also needs to be addressed. This would require additional code if G2H is not used. In fact, the 
scene graph (containing objects) with the cursor and the tissue textures applied to the objects all 
require additional programming to generate the new haptic virtual environment, even if the 
graphical environment is already present. 

(d) All of the pending and rejected claims (1-5) include the limitation creating a cursor with 
attributes of movement within multiple layers, which, the instant Action asserts is taught by Ross 
at col. 9 lines 33-35, figs. 9A-B, and col. 11, lines 20-64. Applicants respectfully assert that this 
is a misreading of Ross. 

First, the Action implies that an object created by Ross has multiple layers. Section III of 
Ross that describes CYBERSCALPEL (col. 8, line 32 to col. 11, line 19, especially col. 9 lines 
14-42) and that relates to Fig. 8 discloses a method of using the virtual cutting tool with flat 
bones (col. 3, 11. 53-54, col. 9, 11. 14-15). Cutting flat bone creates inner and outer layers for the 
cut piece, however, Ross discloses connecting the layers to create a 3D solid with no objects 
inside , (col. 9. 11. 29-42, Figs. 9A-D; col. 10, 11. 15-20) Thus, Ross cannot logically cannot create 
a cursor with attributes of movement within multiple layers of graphic display to create or modify 
one or more virtual objects, since such a cursor would need to be created inside a virtual object. 
This is in contrast to the presently claimed invention, supported by Applicants 1 specification 
Figures 4 and 5, wherein multiple layers, with each layer representing a surface of an object that 
may be inside (i.e., within) another (outer) object. In the exemplary Figure 5, it can be seen that 
the inner layer actually represents a completely different object. The virtual human breast object 
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(outer layer) includes a tumor (inner layer) as an inner object that is independently represented 
and can be touched through the breast. Ross therefore fails to teach or suggest this limitation and 
the present invention as recited in claims 1-5. 

Secondly, with respect to the Action's assertion that col. 1 1. 11. 20-64, discloses creating 
or modifying one or more virtual objects "in real-time by using the force-feedback (haptic) 
devices and technology", Applicants again respectfully disagree with the conclusion reached. 
The virtual collaborative clinic component (VCC) of Ross patent is a networking component that 
enables multiple users to interact in real-time with the same stereoscopic 3D data set that is 
generated by the Mesher and the Cyberscalpel (see the last five lines of the Abstract; Figs. 2, 
12A-B, 13-15, 18; col. 4, 11. 55-58; col. 5, 11. 18-27; col. 10, line 36 to col. 11, line 2; col. 11, line 
20 to col. 16, line 53; and specifically col. 1 1 lines 22-25) It is unclear how reference to the VCC 
leads to the conclusion stated. The only remotely related element Applicants can identify is that 
of simultaneous viewing. The presently claimed invention does allow viewing of four different 
views simultaneously, however, the VCC of Ross simultaneously distributes the same single 
view of an object being manipulated to geographically distributed users (Figs. 1, 5, 9A-D, 1 1 A-E) 
but does not provide a user interface with four different simultaneous views. Applicants fail to 
comprehend the VCCs relevance to the pending patent claims, since the presently claimed 
invention is not directed to networking at all. 

Another significant limitation present in claims 1-5 and not taught or suggested by Ross is 
that a user can create said haptic virtual environment without writing any computer code. For 
example, once a poly-mesh based graphics virtual environment exists (again, Ross teaches how to 
create such an environment for solid 3D, but not multi-layered objects), code will need to be 
generated for developing the haptic processing to make the graphics objects touchable. This code 
typically includes: (1) acquisition of physical properties of the objects; (2) modeling these 
properties into a set of variables to be used by the haptic processing engine so the objects can be 
touched and felt realistically; (3) interface to algorithms for collision detection and computing 



Pflfrp. 7 of n 



Appl. Ser. No.: 09/844,635 

Atty. Docket No.: 12001-105 

Reply to Office Action dtd. 05 May 2005 

force feedback; (4) creation of a 3D object to be used as a symbolic representation, a cursor, for 
the tip of the haptic device position in the virtual space; (5) initialization of the servo loop of the 
haptic device; (6) interface the haptic device to the virtual environment; (7) creating a 
homeomorphism between the physical work space, that is used to move the tip of the haptic 
device, and the virtual work space; (8) development of inter-process communication and feedback 
loop synchronization between the graphic (either mono or stereoscopic executing at usual rates of 
30Hz or 60Hz) and haptic (executing usually at 1000 Hz) processes; (9) applying the computed 
force feedback to the haptic device; and (10) testing and modifying the virtual environment to 
ensure real-time stability of the created haptic objects. It is through this applied force to the 
haptic device that a user holding the device feels the physical properties of objects being touched. 

The presently claimed Graphics-to-haptics (G2H) tool, as described in Applicants 1 
specification, provides a framework for developing such stable haptic applications. The G2H 
uses a real-time visual software development process to automatically convert multi-layers of 
poly-mesh objects into haptic objects with no additional programming (needed for these steps 1- 
10). The G2H automatically extracts information from poly-mesh objects (e.g. geometry, visual 
properties, positions and orientations) in order to construct the haptic scene graph and to make 
the objects in the scene graph touchable. A default haptic texture consisting of physical 
properties such as stiffness, damping, and static and dynamic friction is initially applied. The 
G2H provides a material editor that can generate or modify haptic textures while permitting to 
incorporate new set of properties. Users can interactively modify the values of the properties for 
the textures and save them into a library. Objects feel different as soon as new or modified haptic 
textures are applied to objects. 

(e) The Action also asserts that col. 1 1, 11. 20-64, discloses creating or modifying one or more 
virtual objects "in real-time by using the force-feedback (haptic) devices and technology.' 1 The 
cited section includes the following text: 
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"The VCC component 22 (FIG. 2) is an extension of the reconstruction unit 21 that 
generates a virtual environment that enables multiple users to interact in real-time with the 
same stereoscopic 3D data set. One advantageous use for such a system is to examine 3D 
reconstructions of medical data for consultation and diagnosis by medical practitioners 
who are located in different parts of the world. Thus, the VCC environment is 
particularly suited to displaying 3D models of anatomical reconstructions used in medical 
diagnoses and treatment planning, and the collaborative aspects allow physicians at 
different geographical sites to manipulate these objects as though looking at a common 
display.' 1 

Little to none of the remote interaction and networking capabilities of the VCC appear to be 
directly related to the presently claimed haptic virtual environment tool. Col. 1 1, 11. 20-25 
describes that the VCC component enables multiple users to interact in real-time with the same 
stereoscopic 3D data set, and thus considers only the visual (and thus only graphics) aspect of 
the virtual environment. No consideration of the haptic aspect is provided. The cited section of 
Ross includes the sentence "Development of a virtual environment, imaging and the force- 
feedback (haptic) devices and technologies will make it possible to send sonically scanned data 
back to Earth for visualization, when necessary." This is the only reference to "haptics" in Ross, 
and notably it is referenced only with respect to visualization and not for the sense of touch, as 
in G2H. Ross is devoid of any disclosure relating to the development of haptic virtual objects 
such as algorithms required for collision detection or for generating the force-feedback. It is 
essential to address such issues in creating complex and precise haptic virtual objects that provide 
a realistic sense of touch (as discussed on page 3, paragraph 5 of Applicants 1 specification.) 

Note that in the software development process, the haptic virtual environment is created 
independently of its graphics representation, as described in the paper "Development of 
Stereoscopic-Haptic Virtual Environment" incorporated by referenced in Applicants' 
specification. The collision detection and force feedback computation algorithms required for 
graphics and haptics integration process are addressed on page 3, paragraph 5 of Applicants' 
specification. Note also that there is a distinction between Ross' VCC technologies that make it 
possible to remotely network for visualization, and "haptics" that provide a calibrated sense of 
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touch using a force-feedback (haptic) device. Haptic devices, as used in Applicants* application, 
refers to a device providing feedback that is proportional to the properties of virtual objects being 
modeled. The presently claimed invention makes it possible to differentiate tissue properties of 
computer generated objects, a feature that makes the additional complexity of the software 
worthwhile. Ross fails to teach or suggest feeling the objects existing in a virtual environment. 

Thus, Ross teaches attributes only for the purpose of visualization. The instant 
applications' physical properties and haptic attributes (e.g., stiffness, static and dynamic friction, 
etc.) are important in producing a sense of touch in the virtual objects, (see Fig. 1; page 2, par. 2; 
page 6, par. 18; page 7, par. 19; page 7, par. 21; page 8; par. 23.) Such attributes are not 
addressed anywhere in Ross. Modeling these physical properties into a virtual environment 
require completely different software development techniques compared to modeling visual 
attributes. Being able to apply these physical properties and their variations in real-time using 
the G2H interface software makes it possible for us to "to create one or more virtual objects in 
real-time by using the force-feedback (haptic) devices and technology. " 

The Action also asserts that Ross teaches selecting a virtual object with said cursor, citing 
col. 1 1, 11. 20-64 describing the VCC. As noted above, Ross does not provide a method of 
creating haptic virtual objects. It is respectfully submitted that selection of Ross* objects with a 
cursor, or placement of the cursor on the objects, do not make the objects haptic. The mention in 
Ross of a cursor is at col. 6, 11. 17-20, regarding the pointing device 36 of Fig. 3. The only use of 
Ross* cursor is to indicate a specific location within the image of the 3D object displayed on a 
monitor. Thus, Ross fails to teach or suggest selecting any object, haptic or graphic, with a 
cursor. In contrast, in the presently claimed invention the cursor embodies a means for 
interfacing, initializing, and using a haptic device, (see Fig. 5; pars. 16, 18 and 19.) Applicants 1 3D 
cursor is created as a symbolic representation for the position of the tip of the haptic device in 
the virtual work space that corresponds to the position of the tip of the haptic device. In Figure 
5, the cylinder is an embodiment of a cursor. With the G2H user-interface, any 3D object can be 
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selected as a cursor (see par. 19.) As the haptic device moves in the physical space, so does the 
cursor in the virtual space. If the cursor occupies the same position as one of the virtual objects 
in the haptic scene, a collision has occurred. At this point a user has touched the virtual object 
and the haptic process provides a force feedback to the user via the haptic device. At this point a 
user can feel the physical properties (such as smoothness, hardness, etc) of the virtual object as if 
it was real. It is in this sense that the presently claimed invention can select a virtual object with 
the cursor. This transformation of the graphics virtual object into a touchable (haptic) virtual 
object is processed trough a code module of G2H and thus does not require writing additional 
code. 

Ross' cursor does not represent the tip of a haptic device. Rather, the word "cursor" is 
used in its common meaning of a pointer overlaying a graphical image, and since it has no 
connection to haptics, there is no discussion of collision detection or computing force feedback. 

(f) With respect to claim 5, paragraph 19 of Applicants' specification reads: "The user 
creates a cursor and selects an object. The user places the cursor name in the text dialog box and 
activates a "get cursor" command button. The object selected appears in the "Object Properties 
List Box" where the user can select and modify each object by providing means for creating a 
volumetric 3D object with internal layers. The user can modify the surface stiffness and/or add 
static and dynamic surface friction to any of the layers. In this way a volumetric object is created 
which provides for a realistic touch so that when the user activates the haptic device button, the 
user can "feel" the object." This paragraph describes a method of assigning and changing the 
physical attributes of the layered objects within the virtual environment. Note that when the 
graphics scene contains only solid objects, the same method is used to manipulate the physical 
attributes of these solid objects. The textures and the shapes of the objects being felt are based on 
the physical properties applied to them and the shapes of the objects themselves. This method 
of assigning and changing the physical attribute that creates plurality of virtual objects, as any 
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change in a physical property makes the object feel different. Ross does not disclose applying 
and modeling the physical properties of virtual objects. 

(g) With respect to claims 3-5 (and in a similar manner claims 1-2), the Action asserts that 
col. 1 1, 11. 35-43 of Ross teaches "modifying a surface stiffness of one or more layers of said 
poly-mesh form (by using the force-feedback devices to interact with the virtual object." 
Applicants can find no disclosure in Ross of haptic objects of any kind (nor even a single 3D 
solid haptic object), much less the process of creation of haptic objects. Ross therefore has no 
need to address issues of applying any physical properties such as surface stiffness or static and 
dynamic friction which are inherent to haptics. These concerns are not considered anywhere in 
Ross. The Action appears to base the rejections on the assumption that Ross teaches haptics, 
however this is simply not the case. Ross does disclose creating a graphics virtual environment 
with 3D solid objects (represented as poly-mesh) and sharing it collaboratively with multiple 
users. Ross, however, describes only graphic attributes of objects, and not physically based 
properties of haptics, such as surface stiffness (or static and dynamic friction). Integration of 
these attributes requires very different techniques. 

(h) The comments above with respect to Ross* failure to teach or suggest the limitations 
related to . . .providing a haptic virtual environment... a cursor with attributes of movement within 
multiple layers... means for creating, modifying and saving haptic properties of said one or more 
virtual objects... without writing any computer code are equally applicable to a discussion of 
claims 1-2, and are thus not repeated here. 

With respect to claim 1, the Action additionally asserts that the limitation ...means for 
generating a haptic representation of said one or more virtual objects directly from graphical 
representation of said one or more virtual objects, wherein said one or more virtual objects 
comprise a plurality of layers that are represented by a three-dimensional poly-mesh form is 
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disclosed by Ross' MESHER, discussed at col. 6, 11. 43-65, and col. 1 1, 11. 20-64, and illustrated 
in Figs. 5 and 9A-D. Applicants respectfully disagree. Ross discloses in MESHER a means to 
generate surface models of 3D objects, represented by poly-mesh, and thus graphics 3D solid 
objects from a volumetric image data set, and provides a means for cutting an object and 
connecting the outer and inner surfaces (termed layers) of the same object to convert it to a 3D 
solid object. This is not the same concept as layers created by individual independent poly-mesh 
objects inside another, as recited in claims 1-2. In Applicants 1 system, a layer representing an 
object also has outer and inner surfaces. 

(i) In light of the foregoing, Applicants respectfully submit that Ross fails to teach or 
suggest the inventive methods recited in claims 1-5, and therefore request reconsideration and 
withdrawal of both the § 102(e) and § 103(a) rejections of said claims. Applicants respectfully 
submit that claims 1-5 are now in a condition for allowance, and a notice to that effect is 
earnestly solicited. If any questions arise during the review of this amendment/reply, the 
Examiner is invited to contact the undersigned at (617) 854-4000 to discuss any issue. 



Respectfully submitted, 



BHARTI TEMKIN, et al., 
Applicants 



Date: September 6, 2005 
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